kourouklidesfandomcom-20200213-history
Optimization
This page contains resources about Mathematical Optimization, Computational Optimization and Operations Research. More specific information is included in each subfield. Subfields and Concepts See Category:Optimization for some of its subfields. * Cost Function / Loss Function / Objective Function * Convex Optimization ** Linear Programming *** Big M method *** Simplex Algorithm * Quadratic Programming * Nonlinear Programming ** Karush–Kuhn–Tucker (KKT) conditions * Combinatorial / Discrete Optimization ** Integer Programming ** Dynamic Programming / Optimal Control *** Deterministic Optimal Control *** Stochastic Optimal Control *** Lyapunov Optimization ** Greedy Algorithm ** Travelling Salesman Problem (TSP) ** Approximation Algorithms * Online Convex Optimization ** Mini-Batch Learning * Variational Analysis‎ ** Calculus of variations * Robust Optimization * Lagrange Multipliers * Online Optimization * Iterative Methods ** Powell's Method / Powell's Conjugate Direction Method ** Nelder–Mead Method / Downhill Simplex Method / Amoeba Method ** Constrained Optimization by Linear Approximation (COBYLA) ** Expectation-Maximization (EM) Algorithm ** Levenberg–Marquardt Algorithm ** Iteratively Reweighted Least Squares ** Nonlinear Least Squares ** Ordinary Least Squares / Linear Least Squares ** Weighted Least Squares ** Sequential Quadratic Programming (SQP) / Sequential Least Squares Programming (SLSQP) ** Gauss–Newton Algorithm ** Subgradient Methods (used for Convex Minimization) ** Gradient Descent / Steepest Descent ** Krylov Subspace Methods *** Conjugate Gradient Method *** Biconjugate Gradient stabilized (BiCGSTAB) Method *** Arnoldi Method *** Lanczos Method *** Generalized Minimal Residual (GMRES) Method ** Broyden–Fletcher–Goldfarb–Shanno (BFGS) Algorithm ** Limited-memory BFGS (L-BFGS) ** Truncated Newton Methods / Hessian-free Optimization ** Resilient Backpropagation (Rprop) * Bayesian Optimization * Stochastic Optimization ** Gradient Descent Methods (either full-batch or mini-batch or both) *** Stochastic Gradient Descent (SGD) *** Stochastic Gradient Descent with Cyclical Learning Rates (using Triangular Policy) *** Stochastic Gradient Descent with Restarts (SGDR) / Cyclic Cosine Annealing *** Stochastic Weight Averaging (SWA) *** SGD with Momentum *** Averaged SGD *** AdaDelta *** AdaGrad *** Adam *** RMSprop *** Nesterov’s Accelerated Gradient (NAG) Descent *** NAdam (NAG/Nesterov Adam) *** Projected Gradient Descent *** Particle Mirror Descent (PMD) *** Regularized Dual Averaging (RDA) *** Follow the regularised leader (FTRL) *** Online Gradient Descent *** Adaptive Online Gradient Descent *** Natural Gradient Descent ** Stochastic Gradient Fisher Scoring ** Stochastic Gradient Langevin Dynamics (SGLD) ** Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) ** Stochastic Gradient Riemann Hamiltonian Monte Carlo (SGRHMC) ** Stochastic Gradient Markov Chain Monte Carlo (SGMCMC) ** Stochastic Gradient Nose-Hoover Thermostat (SGNHT) ** Relativistic Stochastic Gradient Descent / Relativistic Monte Carlo ** Stochastic Approximation *** Robbins-Monro Algorithm (using noisy estimates of the gradient) ** Metaheuristics *** Population-based search **** Evolutionary Algorithms **** Genetic Algorithms **** Swarm Intelligence *** Single point search / Trajectory methods / Single-state methods **** Hill-Climbing **** Simulated Annealing **** Tabu search **** Explorative search methods ***** Greedy Randomized Adaptive Search Procedure (GRASP) ***** Variable Neighborhood Search (VNS) ***** Guided Local Search (GLS) ***** Iterated Local Search (ILS) * Inverse Problems ** Regularization *** L1-regularization / Least absolute shrinkage and selection operator (LASSO) *** L2-regularization / Ridge Regression / Tikhonov Regularization *** Early Stopping *** Total Variation (TV) Regularization *** Dropout ** Stochastic Simulation / Monte Carlo Methods * Multi-Objective Optimization / Multicriteria Optimization / Pareto Optimization ** Multi-Objective Linear Programming Online Courses Video Lectures * Optimization Course by Michael Zibulevsky * Convex Optimization I by Stephen P. Boyd * Convex Optimization II by Stephen P. Boyd * Discrete Optimization by Professor Pascal Van Hentenryck - Coursera * Linear and Discrete Optimization by Friedrich Eisenbrand - Coursera * Advanced Optimization and Randomized Methods by A. Smola and S. Sra * Optimization, Learning and Systems by Martin Jaggi * Approximate Dynamic Programming Lectures by Dimitri P. Bertsekas * Optimization Lecture Notes Introductory * Practical Optimization: A Gentle Introduction by John W. Chinneck - a very introductory course * Lecture Notes on Optimization by Pravin Varaiya * Lecture Notes on Engineering Optimization by Fraser J. Forbes and Ilyasse Aksikas * Optimization by Geoff Gordon and Ryan Tibshirani * Optimization: An introduction by A. Astolﬁ * Optimization for Differential Equations by Martin Berggren and Krister Svanberg * Quantitative Methods by Michael Trick * Operations Research. Linear Programming by Ana Zelaia Jauregi Specialized * Convex Analysis and Optimization by Dimitri Bertsekas * Combinatorial Optimization by Santosh Vempala * Discrete Optimization for Vision and Learning by Nikos Komodakis and Pawan Kumar * Lectures on Modern Convex Optimization by Aharon Ben-Tal and Arkadi Nemirovski * Numerical Optimization by Christopher Griffin * Lecture Notes on Continuous Optimization by Kok Lay Teo and Song Wang * Introduction to Online Optimization by Sébastien Bubeck * Lecture Notes on Online Learning by Alexander Rakhlin * EECS260 Optimization by Miguel Á. Carreira-Perpiñán * Optimization Techniques in Engineering by Alan R. Parkinson and John D. Hedengren * Multidisciplinary System Design Optimization by Olivier de Weck and Karen Willcox * ORF523: Advanced Optimization by Sébastien Bubeck * Dynamic Programming and Optimal Control by Raffaello D'Andrea * An Introduction to Inverse Problems by Colin Fox, Geoff K. Nicholls and Sze M. Tan * Inverse Problem in Imaging by Simon Arridge Books Introductory * Chong, E. K., & Zak, S. H. (2013). An Introduction to Optimization. John Wiley & Sons. * Luenberger, D. G., & Ye, Y. (2008). Linear and Nonlinear Programming. Springer. Specialized * Bertsekas, D. P. (2017). Dynamic Programming and Optimal Control. 4th Ed. Athena Scientific. * Bertsekas, D. P. (2016). Nonlinear Programming. 3rd Ed. Athena scientific. * Hazan, E. (2015). Introduction to Online Convex Optimization. Foundations and Trends® in Optimization, 2''(3-4), 157-325. * Bertsekas, D. P. (2015). ''Convex Optimization Algorithms. Athena Scientific. * Puterman, M. L. (2014). Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons. * Fletcher, R. (2013). Practical Methods of Optimization. John Wiley & Sons. * Shalev-Shwartz, S. (2011). Online Learning and Online Convex Optimization.Foundations and Trends®'' in Machine Learning'', 4''(2), 107-194. * Sra, S., Nowozin, S., & Wright, S. J. (2012). ''Optimization for machine learning. MIT Press. * Luke, S. (2009). Essentials of Metaheuristics. Raleigh: Lulu. * Press, W. H. (2007). Numerical Recipes 3rd edition: The Art of Scientific Computing. Cambridge University Press. * Nocedal, J., & Wright, S. J. (2006). Numerical Optimization. Springer. * Ruszczyński, A. (2006). Nonlinear Optimization. Princeton University Press. * Boyd, S. P., & Vandenberghe, L. (2004). Convex Optimization. Cambridge university Press. *Nesterov, Y., & Nesterov, I. E. (2004). Introductory Lectures on Convex Optimization: A Basic Course. Springer. *Saad, Y. (2003). Iterative Methods for Sparse Linear Systems. Siam. *Vogel, C. R. (2002). Computational Methods for Inverse Problems. Siam. *Kelley, C. T. (1999). Iterative Methods for Optimization. Siam. *Dennis Jr, J. E., & Schnabel, R. B. (1996). Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Siam. *Rustagi, J. S. (1994). Optimization techniques in statistics. Elsevier. *Scales, L. E. (1985). Introduction to non-linear optimization. Springer-Verlag. Software See List of Optimization Software for the complete list. * Optimization Toolbox - MATLAB * Optimization (scipy.optimize) - Python * climin - Python * PyOpt - Python * dlib - C++ (with Python API) * CVXPY - Python * CVX - MATLAB * libLBFGS - C * PyLBFGS - Python * NLopt - C, C++, Fortran, Matlab or GNU Octave, Python, GNU Guile, Julia, GNU R, Lua, OCaml * hessianfree - Hessian-free Optimization for Deep Networks * MOSEK Fusion API - Python framework for conic optimization * MOSEK - C, Java, MATLAB, .NET, Python, R * Gurobi - various languages See also * Sparse Coding * Kernel Methods * Estimation Theory * Machine Learning * Computer Vision * Computational Finance Other Resources *Mathematical Optimization - Google Scholar Metrics (Top Publications) *NEOS Optimization Guide *COIN-OR - Computational Infrastructure for Operations Research *Decision Tree for Optimization Software Links to optimization source codes *Mathematical Programming Glossary *Optimization Related Links *Optimization - Notebook *Stochastic Approximation - Notebook *Convex Optimization - Metacademy *OPTML - NIPS Workshop on Optimization for Machine Learning *Metaheuristics and Local Search *A good book for Inverse Problems - Math StackExchange *A Global Optimization Algorithm Worth Using (for hyperparameters) - blog post in dlib *opt (GitHub) - code *Linear Regression – Theory - blog post *Gradient Descent - blog post *Implement Gradient Descent in Python (Medium) - blog post *Visualizing the gradient descent method - blog post Category:Optimization Category:Algorithms